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SUMMARY 


OBJECTIVE 

This  Technical  Document  (TD)  presents  guidelines  for  the  timely 
implementation  of  on-line  system-level  test  and  performance  monitoring 
capability  into  the  design  of  military  prime  systems.  The  presentation  of 
material  is  intended  to  stimulate  detailed  inquiries  into  substantive 
qualitative  and  quantitative  issues  concerning  "state-of -health"  of  a 
system,  in  proper  proportion  with  other  design  issues  and  requirements 
such  as  off-line  test,  maintenance  and  reliability,  and  logistic  support. 


APPROACH 

The  data  in  this  TD  is  presented  in  flow  chart,  text,  and  checklist 
format  as  guidelines  to  program  managers,  engineers  and  technical  staff. 
Interrelationship  is  maintained  between  flow  chart  numbered  items  and  text 
paragraphs,  for  ready  cross  reference.  As  such,  the  text  provides  in- 
depth  explanation  of  the  flow  charts,  and  the  check  list  data  provides 
extra  assurance  that  key  performance  monitoring  issues  will  be  considered. 
The  entire  presentation  is  normal  to  the  time-sequence  flow  of  the  DSARC 
process  for  major  system  acquisition.  Although  the  prime  thrust  of  this 
TD  is  directed  toward  requirements  for  monitoring  the  "health"  of  a 
system,  it  is  not  intended  to  overshadow  the  importance  of  off-line 
testing  requirements .  As  such,  program  managers  and  engineers  are  urged 
to  constantly  seek  a  balance  between  on-line  and  off-line  interests, 
toward  elimination  of  false  indications  and  testing  ambiguities,  resulting 
in  the  most  reliable,  useful  and  valuable  mix  of  test  and  monitoring 
techniques. 
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INTRODUCTION 


Each  major  system  acquisition  program  has  its  unique  features. 
Differences  in  time,  cost,  technnology,  management,  and  contracting 
approach  must  be  recognized.  However,  despite  the  differences,  the  basic 
acquisition  process  is  common  to  all  programs,  being  based  on  DoD 
Directive  5000.1  including  0MB  Circular  A-109,  both  titled  "Major  System 
Acquisitions."  Commonality  is  also  apparent  with  DoD  Instruction  5000.2 
"Major  System  Acquisition  Procedures."  Figure  1  illustrates  the  basic 
acquisition  process  or  cycle. 

The  principal  activities  in  the  major  system  acquisition  process  are 
iterative.  As  more  knowledge  of  needs,  alternative  solutions,  actual 
capabilities,  resources,  and  priorities  is  acquired,  some  steps  in  the 
overall  cycle  may  be  repeated,  to  permit  decisions  to  be  made  in  a  total 
system  context.  One  such  principal  activity,  system-level  test  and 
performance  monitoring  i.e.  those  functions  involved  in  the  determination 
of  the  "state  of  health"  of  a  system  during  operation,  is  emphasized  in 
this  technical  document.  The  flow  charts  depicting  the  Defense  Systems 
Acquisition  Review  Council  (DSARC)  acquisition  cycle  illustrate  primarily 
those  activities  that  have  interface  or  association  with  the  performance 
monitoring  activity.  Program  managers  should  exercise  judgement  and 
flexibility  to  "tailor"  their  programs  according  to  unique  features  or 
limitations  imposed. 


o 


Figure  1.  Major  System  Acquisition  Cycle 


PRE-MILESTONE  0 


Activities  in  the  Pre-Milestone  0  phase  of  the  acquisition  process 
identify  and  document  a  mission  need.  Its  purpose  is  to  summarize  the 
requirements  in  preparation  of  gaining  Secretary  of  Defense  approval  to 
explore  the  concept  further.  It  also  leads  to  the  designation  and 
appointment  of  a  program  manager.  The  flow  of  activity  is  depicted  in 
Figure  2  located  at  the  end  of  Section  1. 


DETERMINATION  OF  MISSION  NEEDS.  DoDD  5000.1  and  0Mb  Circular  No  A-109 
require  each  DoD  agency  to  conduct  a  continuing  analysis  of  current  and 
forecasted  mission  capabilities,  technological  opportunities,  overall 
priorities,  and  resources.  From  this,  are  determined  the  needs  as  related 
to  the  ability  to  perform  the  operational  mission.  Needs  are  described  in 
terms  of  the  mission,  purpose,  capability,  operating  constraints,  and 
schedule  and  cost  objectives.  This  action  presents  the  first  opportunity 
to  identify  system-level  test  and  performance  monitoring  objectives  to  be 
brought  into  focus  as  system  requirements  in  subsequent  acquisition  cycle 
phases. 


OPERATIONAL  REQUIREMENT  (ODDI  5000.2).  In  recognition  that  a  capability 
to  monitor  system  performance  during  operation  increases  mission 
effectiveness  in  terms  of  enhanced  maintenance  capability  and  greater 
command  decision  visibility,  the  requirement  for  performance  monitoring 
capability  should  be  seriously  considered  for  inclusion  in  the  operational 
requirement.  In  such  circumstances,  applicable  conceptual  performance 
monitoring  goals  and  parameters  should  be  indicated  in  the  statement  of 
operational  requirement. 


PREPARE  AND  APPROVE  THE  MISSION  ELEMENT  NEED  STATEMENT  (MENS)  (DODD 
5000.1,  POO I  5000.?.)  When  the  analysis  of  current  and  forecasted  mission 
capabilities  identifies  a  deficiency  in  existing  agency  capabilities,  or 
an  opportunity  to  establish  new  capabilities  in  response  to  a 
technological  advantage,  such  deficiency  will  be  formally  set  forth  in  a 
mission  need  statement.  As  such,  the  Mission  Element  Need  Statement 
(MENS),  described  in  DoDI  5000.2  Enel.  2,  is  completed  by  the  cognizant 
Naval  Material  Command  (NAVMAT)  or  Systems  Command  (SYSCOM)  component. 
Its  approval  by  the  Secretary  of  Defense  (SECDEF)  is  authority  to  proceed 
to  the  next  acquisition  phase. 

Constraints  of  historical  origin  that  should  apply  to  any  acceptable 
solution,  will  be  stated  initially  in  the  MENS.  Examples  of  such 
constraints  are  operational,  logistics  and  design  considerations,  resource 
limits,  and  timing  requirements.  Performance  monitoring  requirements  will 
seldom  be  the  singular  motivation  for  the  acquisition  of  a  new  system. 
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However,  any  performance  monitoring  contribution  to  improved  mission 
effectiveness  warrants  consideration  as  a  performance  requirement  in  the 
"Assessment  of  Need"  portion  of  the  MENS  by  citing  the  inadequateness  of 
system  readiness  without  it,  and  the  resultant  degradation  of  mission 
performance.  Approval  of  the  MENS  constitutes  the  milestone  decision  by 
SECDEF  to  proceed  into  Phase  1,  concept  exploration,  to  identify  and  to 
explore  alternative  solutions.  It  does  not  commit  to  a  particular 
solution  at  this  point. 


SECTION  1 


CONCEPT  EXPLORATION  PHASE 


1.0  The  concept  exploration  phase  of  the  acquisition  process  provides 
commitment  only  to  identify  and  to  explore  alternative  solutions.  A  broad 
base  of  qualified  manufacturers  is  sought  initially,  to  provide  the  widest 
range  of  alternative  solutions  capable  of  satisfying  the  mission  need. 
These  are  subsequently  reduced  to  a  selected  number  who  offer  best 
technical  and  cost-effective  solutions  of  the  mission  need.  The  flow  of 
activity  for  this  phase  is  depicted  in  Figure  2  at  the  end  of  this 
section. 


1.1  INITIAL  PROGRAM  ACQUISITION  STRATEGY  (DODD  5000.1,  DODI  5000.2). 

Development  of  the  initial  program  acquisition  strategy  is  usually 
accomplished  in  broad  terms  by  NAVMAT  or  System  Commanders  as  soon  as 
possible  after  Milestone  0.  It  is  a  reflection  of  the  management  concepts 
to  be  used  subsequently  by  the  program  manager  in  directing  and 
controlling  all  elements  of  the  acquisition,  in  response  to  specific  goals 
and  objectives  set  to  assure  satisfaction  of  the  approved  mission  needs. 


1.2  ASSIGN  PROGRAM  MANAGER  (PM)  (DODD  5000.1,  OPNAVI  5000.1).  A  program 
manager  is  designated  by  the  Chief  of  Naval  Material  or  a  Systems  Command 
as  soon  as  possible  after  the  Milestone  0  decision.  Program  objectives 
are  developed  that  set  forth  the  capability  (in  mission  needs,  not 
equipment  solution  terms)  cost,  and  schedule  goals  being  sought  in  the 
system  acquisition  program.  These  objectives  are  incorporated  in  a 
written  charter,  which  defines  the  authority,  responsibility,  and 
accountability  of  the  program  manager. 


1.3  RECRUIT  STAFF/TAILOR  ACQUISITION  STRATEGY  (DODD  5000.1,  DODI 
5000.27^  An  initial  responsibility  of  the  program  manager  is  to  recruit  a 
staff,  or  to  identify  a  team  with  the  requisite  skills  and  experience  to 
manage  the  assigned  system.  Another  of  the  program  manager's  first  tasks 
is  to  develop,  with  his  staff,  an  efficient,  effective  and  economical 
acquisition  strategy  encompassing  the  technical,  business,  and  management 
functions  of  the  entire  acquisition  process.  This  strategy  forms  the  basis 
for  solicitation  requirements  for  system  level  alternatives,  as  well  as 
subsequent  acquisition  and  management  plans.  Initially  limited,  it  is  be 
broadened  and  refined  as  the  program  progresses.  Table  1  contains  a 
checklist  of  on-line  test  and  monitoring  items  for  consideration  in  the 
development  of  acquisition  strategy. 
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Have  the  following  on-line  test  and  monitoring  aspects  been 
considered  in  the  development  of  the  program  manager's  acquisition 
strategy? 


a.  Has  a  staff  member  been  identified  to  assure  coverage  of  the 
performance  monitoring  aspects? 

b.  Have  the  following  performance  monitoring  strategies  been 
considered? 

•  Identification  of  overall  performance  monitoring  goals 

•  Impact  on  design,  i.e.,  compatibility  with  other 
disciplines  such  as  R&M,  accessibility,  safety,  support, 
etc. 

•  On-line  and  off-line  diagnostics  requirements 

•  Impact  on  support,  i.e.,  maintenance,  repair,  parts 

•  Identification  of  development  and  operational  testing 
requirements 

•  Configuration  management  applications 

c.  Are  overall  testability  aspects  constrained  by  the  operational 
mission,  i.e.,  as  with  an  orbiting,  unmanned  space  craft,  thus  requiring 
special  performance  monitoring/self-test/fault  tolerance  consideration? 

d.  Is  requirement  for  auto-calibration  capability  contemplated? 


Table  1.  Acquisition  Strategy  Checklist 
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1.3.1  Design  Model.  Program  Management  Office  (PMO)  design  and  test 
engineering  personnel  will  develop  a  conceptual  design  model  using  mission 
needs  as  a  basis,  and  applying  the  proven  precepts  of  such  disciplines  as 
maintainability,  reliability,  availability,  human  engineering,  and  safety. 
Appropriate  criteria  of  testability,  which  is  emerging  as  a  stand-alone 
design  discipline  and  which  includes  on-line  test  and  monitor'  g  concepts, 
should  also  be  considered  in  the  design  model  development. 

Constraints  due  to  the  unique  nature  of  the  mission  can  greatly  restrict 
the  testability  possibilities,  and  must  be  considered  in  order  to  fit  the 
conceptual  model  into  the  operational  requirement  framework.  Such 
constraints  might,  for  example,  be  imposed  by  a  requirement  for  unusually 
long  hours  of  unattended  system  operation. 

Severe  restrictions  on  weight,  form,  fit,  power  consumption,  etc.  might 
preclude  the  inclusion  of  additional  components  or  hardware  partitioning 
which  would  improve  testability/accessibility.  For  example,  an  unmanned 
spacecraft  will  require  little  or  no  hands-on  testability  designed  into 
it.  It  is  envisioned  that  such  a  vehicle  will  experience  very  long  hours 
of  unattended  operation,  thus  requiring  heavier  than  usual  reliance  on 
built-in  state-of-health  determination  and  self-maintenance  capability. 
Design  considerations  for  attaining  such  a  capability  as  they  apply  to 
fulfillment  of  the  mission  include  self-test  and  fault-tolerance 
techniques.  Self-test  employing  built-in  test  (BIT)  techniques  and 
possibly  built-in  test  equipment  (BITE)  offers  the  capacity  to  detect  and 
possibly  isolate  system  faults  without  the  use  of  external  test  equipment. 
The  implementation  of  BIT  can  take  many  forms,  can  be  cursory,  or  at  great 
depth.  At  this  stage  of  a  program,  the  engineering  personnel  should  be 
concerned  with  the  general  concept:  they  should  consider  the  contribution 
of  self-test  techniques  to  mission  fulfillment,  while  recognizing 
constraints  as  described  above  for  implementation  of  testability 
techniques. 

Fault-tolerant  techniques  are  typically  applicable  to  systems  wherein  a 
very  high  degree  of  reliable  operation  is  vital,  as  in  space  missions,  for 
example.  Since  fault-tolerance  techniques  tend  to  involve  considerable 
additional  hardware,  the  natural  system  constraints  must  be  considered 
carefully,  because  the  penalties  for  inclusion  of  such  significant 
add  it  - .  ia 1  circuitry  or  hardware  may  rapidly  push  a  concept  beyond  the 
realm  of  feasibility.  The  great  strides  being  made  in  the  scale  of 
integration  of  circuits,  however,  allow  for  considerable  amounts  of 
additional  circuitry,  such  as  redundant  integrated  circuit  (IC) 
processors,  memory,  etc,  to  be  employed  at  little  cost  or  penalty. 

The  above-referenced  aspects  of  design  must  be  considerd  in  the  broadest 
sense  to  clarify  fulfillment  of  mission  needs,  while  not  suppressing  the 
creativity  and  competition  aspects  encouraged  in  this  stage  of  the 
acquisition  process. 

Formulation  of  data  for  the  PMO  System  Engineering  Management  Plan  (SEMP) 
should  be  initiated  at  this  time  for  inclusion  in  the  acquisition 
strategy. 
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1.3.2  Support  Model.  Integrated  Logistics  Support  (ILS)  planning  impacts 

upon  and,  in  turn,  is  impacted  by  tne  engineering  activities  throughout 
the  system  life  cvcle.  Toward  development  of  the  support  model,  support 
descriptors  are  fashioned  from  the  criteria  and  constraints  of  the  stated 
mission  needs.  These  descriptors  include:  unique  operational 

constraints,  basing  and  deployment  concepts,  maintenance  concepts,  repair- 
level  concepts,  personnel  and  training  needs,  and  support/test  eqipment 
constraints.  Anticipated  performance  monitoring  requirements  envisioned 
at  this  juncture  in  the  acquisition  process  will  be  considered  in  forming 
the  support  model.  Quantitative  and  qualitative  values  for  ILS 
descriptors  should  be  attempted  as  early  as  possible,  even  to  the  extent 
of  obtaining  these  values  from  recent  past  historical  data. 

In  addition  to  developing  the  basic  support  model,  ILS  activity  by  the  PMO 
in  the  concept  exploration  stage  should  identify  ILS  Plan  requirements. 

Logistic  Support  Analysis  (LSA)  requirements  and  LSA  data  elements  per 
MIL-STD-1388,  and  Level  of  Repair  (LOR)  constraints  per  MIL-STD-1390. 

1.3.3  Initial  System  Test  and  Evaluation  Goals.  A  prime  purpose  of  Test 

and  Evaluation  is  to  assess  the  chance  that  some  element  of  the 
acquisition  process  may  produce  an  unintended  result.  Therefore,  test 
and  evaluation  planning  is  initiated  as  early  as  possible.  During  the 

concept  exploration  phase,  test  and  evaluation  reqirements  are  initiated 
in  the  form  of  broad-based,  development  test  plans  and  test  procedures 
documents.  Emphasis  will  be  placed  on  Development  Test  and  Evaluation 
(DT&E)  requirements  when  appropriate  to  assist  in  the  selection  of 

alternative  system  concepts. 

1.3.4  Configuration  Management  Policy.  The  extent  of  application  of  the 

configuration  management  (CM)  discipline  to  the  system  envisioned  for 
fulfilling  a  mission  need,  should  be  initiated  in  the  concept  exploration 
phase.  This  determines  the  initial  CM  policy  which  will  encompass  basic 
planning  and  procedural  information  for  the  program  manager's  CM  plan. 
This  plan  will  be  updated  periodically  as  the  acquisition  takes  shape. 
The  CM  policy  should  also  provide  the  basis  for  defining  contractor  CM 

plan  requirements. 

CM  actions  on  which  performance  monitoring  may  have  impact,  include: 

•  CM  plan  as  it  relates  to  specifications  preparation 

•  Determination  of  primary  interfaces 

•  Documentation  for  baselines 

•  Configuration  audits 

•  Mission  constraints  requiring  unusual  on-line  testing/monitoring 
requirements 
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1.4  PREPARE  PRELIMINARY  SYSTEM  SPECIFICATION  (MIL-STD-490,  NAVMAT  I 
5000.22Ay7~  Based  on  mission  needs,  a  preliminary  system  specification  is 
prepared  by  the  program  management  office  staff,  defining  the  system's 
operational  requirements  and  environment,  characteristics,  design, 
development,  support  and  test  requirements .  This  specification,  prepared 
in  MIL-STD-490  format,  will  be  provided  to  interested  contractors  as  part 
of  a  Request  for  Proposal  (RFP)  soliciting  alternative  solutions  to  the 
mission  need.  The  purpose  of  the  government-produced  system  specification 
is  to  consolidate,  to  set  a  common  base,  and  to  present  mission  needs.  It 
should  not  diminish  competition  by  constraining  contractor  innovation. 
Contractors  selected  to  continue  into  advanced  stages  of  the  acquisition 
will  be  provided  the  opportunity  to  update  the  preliminary  system 
spec  if icat ion  based  on  their  alternative  solutions. 


The  following  on-line  test  and  monitoring  applications  and  management 
planning  factors  should  be  considered  for  inclusion  in  the  preliminary 
system  specification. 


1.4.1  Incorporate 

On-Line 

Test  and  Monitoring 

Requirements. 

The 

following  specified 

values  i 

(percentages  or  numbers) 

should  be  appropriate 

for  the  concept  at  hand,  and  are  not  considered  as  generally  standardized 
values.  However,  at  this  early  stage  in  the  procurement  cycle,  actual 
values  for  many  of  the  on-line  testability  or  monitoring  parameters  may 
not  be  determinable,  necessitating  use  of  applicable  historical  data  when 
available.  Computer  values  should  be  added  in  the  finalized  system 
specification  as  they  are  determined.  However,  any  of  the  following 
listed  parameter  values  which  are  already  known  and  are  important  to  the 
fulfillment  of  the  basic  mission  need  should  be  included  in  the 
preliminary  system  specification. 

•  False  Alarms  Percentage.  Indicate  maximum  allowable  percentage 
of  fault  indications  which  may  be  false  alarms,  or  how  many 
false  alarms  per  n  hours  of  operation  are  acceptable:  n 
(typically)  equals  one  thousand  or  one  million. 

•  Mean  Fault  Detection  Time.  For  monitoring  devices,  specify 
requirement(s)  on  maximum  times  to  either  detect  or  indicate  a 
fault  once  it  has  occurred. 

•  Mean  BIT  Running  Time.  Specify  maximum  allowable  time  to  verify 
by  means  of  BIT  that,  indeed,  a  failure  has  occurred. 

•  Test  Monitoring  Thoroughness.  Specify  a  percentage  value  of  all 
functions  to  be  monitored.  At  system  level,  monitoring  should 
provide  a  one  hundred  percent  assurance  that  all  major  functions 
are  operating. 

•  Maintenance  Personnel  and  Operator  Skill.  Indicate  that 
utilization  of  low-skilled  operator  personnel  is  made  possible 
by  comprehensive  BIT  and  on-line  performance  monitoring. 


•  Availability.  The  availabilities  of  systems  with  BIT  and  on¬ 
line  monitoring  should  be  specified  to  some  probability  of 
survival.  MIL- STD- 72 1 B  defines  availability  as  a  measure  of  the 
degree  to  which  an  item  is  in  the  operable  and  commitable  state 
at  the  start  of  a  mission,  when  the  mission  is  called  at  an 
unknown  (random)  point  in  time. 

•  Memory  Requirement.  Minimum-allowed  memory  for  BIT  functions 
must  be  stated. 

1.4.2  Integrated  Logistic  Support  Plan  (ILSP).  The  development  of  an  on¬ 
going  system  maintenance  plan  should  be  considered  for  large  systems  which 
will  utilize  a  significant  amount  of  on-line  performance  monitoring 
techniques.  Such  a  plan  would  include: 

•  An  estimate  of  the  man-hours  required  to  perform  the  necessary 
system  maintenance  should  be  made  in  order  to  compute  the  system 
Mean  Time  to  Repair  (MTTR)  and  the  Mean  Time  to  Fault  Locate 
(MTFL)  for  LSA  and  LOR  application. 

•  Any  foreseen,  unique  maintenance  of  performance  monitoring 
requirements  or  problems,  such  as  deployment  of  a  system  in  a 
hostile  environment,  where  special  telemetric  techniques  would 
be  required  to  relay  performance  data  from  the  remote 
application  site  back  to  a  manned  control  center,  should  be 
covered  at  this  early  stage,  if  possible.  This  will  provide  the 
bidding  contractors  time  to  work  out  viable  potential  solutions 
early  in  the  acquisition  cycle,  avoiding  problems  later. 

1.4.3  Test  and  Evaluation  Master  Plan.  Requirements  for  contractor  DT&E 
and  OT&E  plans  should  specify  the  proper  test  environment  to  enable 
testing  of  any  performance  monitoring,  BIT,  or  other  self-test  application 
in  the  system.  The  preferred  technical  approach  to  a  system's  performance 
monitoring,  and  the  necessary  formal  verification  methods  to  be  applied, 
should  be  identified. 

1.4.4  Configuration  Management  Plan  Requirements.  Baselines  are 
established  at  recognized  points  in  a  program  where  it  is  necessary  to 
define  a  formal  control  departure  point  for  any  future  changes  in  either 
performance  or  design. 

Provision  should  be  stated  for  contractors  to  track  and  to  provide 
appropriate  modifications  to  any  self-test/performance  monitoring 
techniques  whenever  changes  to  the  prime  system  are  incorporated.  It  is 
advisable  to  place  monitoring  and  self-check  functions  under  software 
control  to  the  maximum  extent.  This  permits  modifications  to  the 
monitoring/self  check  structure  to  keep  pace  with  changes  to  the  system 
hardware  or  performance,  thus  averting  false  alarm  indications. 
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1.5  PREPARE  RFP  FOR  ALTERNATIVE  SOLUTIONS.  (DODD  5000.1,  DoDI  5000.2). 

The  Request  For  Proposal  (RFP)  should  explain  the  need  in  mission  [not 
equipment)  terms,  schedule  objectives  and  constraints,  program  cost 
objectives,  capability  objectives,  and  operating  constraints.  The 
solicitation,  in  terms  of  mission  need  is  the  key  to  this  portion  of  the 
acquisition  process.  The  contractors  should  be  free  to  propose  their  own 
technical  approach,  main  design  features,  and  alternatives  to  schedule, 
cost,  and  capability  goals.  The  purpose  of  this  type  of  solicitation  is 
to  gain  the  benefits  of  industry  innovation  and  competition,  and  not  to 
introduce  constraint  by  preordained  or  prematurely  selected  equipment 

approaches.  It  should  provide  background  information  on  prior  studies, 

constraints  inherent  in  the  need,  and  technology  developed  by  Government 
laboratories  or  at  Government  expense.  The  accessibility  of  related 

Government  information  should  be  provided  as  part  of  the  solicitation. 
The  solicitation  should  not  restrict  the  contractors  by  specifying  or 
referencing  Government  specifications  and  standards  Instead,  a  type  I 

statement-of-work  (MIL-HDBK  245A),  is  used  which  defines  technical 

requirements  in  terms  of  goals  rather  than  in  quantitative/qualitative 

terms.  Broad  definitions  needed  to  define  the  capability  objectives  of 
the  mission  needs,  including  performance  monitoring  goals  and  interface 
with  other  design  specialties,  are  obtained  from  the  technical  staff 
inputs  to  the  acquisition  strategy.  Additionally,  documents  produced  in- 
house,  such  as  the  System  Engineering  Management  Plan  (SEMP),  Integrated 
Logistics  Support  Plan  (ILSP),  Level  of  Repair  (LOR)  Analysis,  and 

Logistics  Support  Analysis  (LSA),  Test  and  Engineering  Master  Plan  (TEMP) 
and  Configuration  Management  Plan  (CMP),  will  be  initiated  in  this  phase 
of  the  acquisition  process.  Table  2  contains  a  checklist  of  on-line  test 
and  monitoring  items  for  consideration  as  inputs  to  this  RFP.  Table  3 
contains  sample  RFP  task  statements  for  use  as  applicable. 

1.5.1  Pre-RFP  Briefing.  In  the  interest  of  expediting  the  solicitation 
process  for  the  concept  exploration  phase,  program  managers  are  encouraged 
to  conduct  orientation  briefings  for  industry,  and,  where  appropriate, 
allow  industry  to  coinment  on  a  draft  of  the  solicitation  and  the  system 
acquisition  strategy.  Other  objectives  are  to  remove  inhibitors  to 
innovative  solutions  in  response  to  the  solicitation,  and  to  improve 
achievement  of  program  objectives.  The  pre-solicitation  briefing  presents 
the  opportunity  for  discussing,  among  other  issues,  the  on-line  test  and 
monitoring  aspects  of  the  mission  needs.  Any  Government  historical  data 
in  this  line  should  be  made  available  to  contractors  at  this  time.  The 
above  applies  for  post-RFP  bidders'  conferences  as  well. 


1.6  CONTRACTOR'S  CONCEPTUAL  FEASIBILITY  STUD IE S/PROPOSAL  (DODD  5000.1, 
DoDI  5000.2,  SECNAV  I  500O.1).  Solicitations  are  sent  to  a  selected, 
broad  base  of  qualified  firms  to  obtain  the  widest  range  of  alternatives 
capable  of  satisfying  the  mission  need.  Interested  contractors  will 
prepare  technical  and  cost  proposals  based  on  the  RFP  requirements. 
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The  following  on-line  test  and  monitoring  goals  relating  to  mission 
needs  should  be  addressed. 

a.  Does  the  RFP  explain  performance  monitoring  objectives  in  terms 
of  contribution  to  mission  effectiveness  and  life  cycle  cost  (LCC), 
through  improved  maintenance  and  enhanced  command  decision  capability? 

b.  Have  sufficiently  broad,  performance  monitoring  goals  been 
stated,  based  on  the  operational  mission  and  on  foreseen  maintenance 
environment? 

c.  Is  requirement  included  for  the  contractor  to  explain  his 
understanding  of  the  following  performance  monitoring  requisites,  as  may 
be  required? 

•  Compatibility  between  on-line  and  off-line  use  of  test-points, 
diagnostics  and  sensors 

•  Optimum  mix  of  on-line  testing/monitoring,  versus  off-line 
testing  based  on  the  operational  mission 

•  Timing  of  the  performance  monitoring  function  with  system 
operation 

•  Potential  for  performance  monitoring  to  decrease  reliability  and 
increase  testability  problems,  and  how  he  intends  to  preclude 
this 

•  Consideration  of  self-test  and  fault-tolerance  design  principles 

•  Effect  of  performance  monitoring  on  level-of -repair  and  life 
cycle  cost 

•  Effects  of  on-line  test  and  monitoring  capability  on 
acquisition,  ownership,  and  life-cycle  costs 

•  Optimum  mix  of  hardware/software  to  effectively  implement 
performance  monitoring  and  self-test  functions 

•  Scheme  for  accessibility  of  monitoring  and  self-test  functions 
as  applicable  to  hardware  and  software  modularization,  external 
test/control  points,  emulation  capability 


Table  2.  Alternative  Solutions  RFP  Checklist 


a.  The  contractor  shall  describe  his  procedures  for  the  application 
of  the  on-line  test  and  performance  monitoring  technologies  as  they  may 
apply  in  fulfillment  of  the  stated  mission  needs.  As  a  minimum,  the 
following  shall  be  considered: 

•  Identify  the  broad  parameters  for  an  on-line 
testing/monitoring  capability  including 

oo  Self-test 

oo  System  Status  monitoring 
oo  Fault  detection 
oo  Fault  isolation 
oo  Fault  tolerance 

§  Identify  broad  parameters  for  manual  and  automatic  off-line 
testing 

•  Project  an  optimum  mix  of  on-line  test/monitoring  and  off¬ 
line  test  capabilities 

•  Describe  procedures  planned  to  assure  optimum  compatibility 
between  on-line  and  off-line  test  points,  sensors,  and 
diagnostics 

•  Describe  planned  parameters  for  timing  the  on-line,  self¬ 
test/monitoring  functions  with  system  operation 

•  Plan  steps  to  preclude  the  on-line  test  and  monitoring 
functions  from  appreciably  decreasing  reliability  and 
increasing  testing  problems 

b.  The  contractor  shall  indicate  how  he  proposes  to  relate  his 
envisioned  on-line  system  test  and  monitoring  technical  impacts  to  cost 
and  schedule  impacts. 

c.  The  contractor  shall  conduct  and  record  Life-Cycle  Cost  analysis 
data  of  off-line  versus  on-line  test  capability. 

d.  The  contractor  shall  conduct  and  record  Life-Cycle  Cost  analysis 
data  for  use  of  BIT  versus  no  BIT. 

e.  The  contractor (s)  Design  for  Testability  (DFT)  features  shall 
project  level  of  technology,  i.e..  Large  Scale  Integration/Very  Large 
Scale  Integration  (LSI/VLSI)  opto-electrical ,  etc.,  devices  in  broad 
parameters. 


Table  3.  Conceptual  RFP  Task  Statements 
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1.7  PROPOSAL  REVIEW  AND  SELECTION  (DODD  5000.1,  SECHAV  I  5000.1). 

Proposals  are  evaluated  an3  tine  most  promising  concepts  selected  for 
further  exploration.  The  selection  is  based  on  review  and  evaluation  by 
a  team  of  experts  mustered  from  within  and  outside  the  organization 
responsible  for  management  of  the  acquisition  effort.  The  evaluation  must 
consider  the  technical  capability  of  the  concept  to  fulfill  mission  needs, 
including  the  resources  required.  When  it  appears  that  the  need  may 
require  extraordinary  methods  of  system  monitoring  or  on-line  testing 
application,  the  proposal  evaluation  team  should  verify  or  determine  that 
the  proposal  solution  is  consistent  with  the  need  as  stated  in  the  RFP. 

Other  evaluation  considerations,  on  which  on-line  test  and  monitoring 
could  have  impact,  include:  acquisition  and  ownership  costs,  time  to 
develop  and  procure,  and  the  relevant  accomplishment  record  and  competence 
of  key  personnel  of  the  competitors  in  the  self-test/self-repair 
technologies. 

The  purpose  of  the  evaluation  is  to  select  a  number  of  contractors 
providing  the  best  technical  solutions  within  reasonable  costs  that  will 
fulfill  the  stated  mission  need.  Table  4  contains  a  checklist  of  on-line 
test  and  monitoring  items  for  consideration  in  proposal  evaluation. 


1.8  FINALIZE  AND  FORWARD  DECISION  COORDINATING  PAPER.  (DODI  5000.2, 
SECNAV  I  5000.1,  OPNAV  I  SOPOT*^  The  purpose  of  the  Decision 
Coordinating  Paper  ( OCR )  is  to  provide  documentation  for  use  by  the 
Defense  Systems  Acquisition  Review  Council  to  formulate  a  recommendation 
for  the  Secretary  of  Defense  (SECDEF)  Milestone  decisions.  As  major 
contributor  to  the  DCP,  the  program  manager  and  his  staff  initiate  action 
in  conjunction  with  the  Navy's  sponsoring  agency  to  prepare  this  document 
for  the  Milestone  I  decision.  The  format  is  described  in  DoD  Instruction 
5000.2.  Goals  and  thresholds  for  all  aspects  of  projected  on-line 
testability  functions  that  have  potential  to  drive  system  effectiveness 
and  costs  will  be  recorded  in  the  DCP.  Examples  of  these  goals  and 
thresholds  include:  system  status  monitoring,  self-test,  fault  detection, 
and  fault  tolerance.  Annex  A  to  Enclosure  3  of  DoD  Instruction  5000.2 
provides  the  format  for  recording  these,  as  well  as  other  performance 
data. 


1.9  DECISION  COORDINATING  PAPER  (DCP)  APPROVAL  ( DODD  5000.2,  SECNAV  I 

5000.1,  OPNAVI  5OO0.46).  Approval  by  the  SECDEF  Ts  contingent  on  the 
sufficiency  of  the  selected  alternatives  to  fulfill  the  mission  need  at 
reasonable  cost.  SECDEF  approval  of  the  DCP  establishes  Milestone  I  and 
is  approval  for  continued  exploration  of  alternative  concepts  by  the 
successful  contractors. 
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Contractor's  proposals  should  be  assessed  for  degree  of  use  (as 
applicable;  of  the  following  on-line  test  and  monitoring  considerations. 

a.  Does  the  evaluation  team  include  person(s)  adequately  qualified 
in  areas  of  design  for  testability,  or  self-test  and  monitoring? 

b.  Have  the  following  considerations  been  evaluated  considering  the 
impact  of  on-line  test  and  monitoring? 

•  Appropriate  contribution  to  fulfillment  of  mission  needs 

•  Acquisition,  ownership  and  life-cycle  costs 

•  Procurement  time 

c.  Have  broad  parameters  been  outlined  for  the  following 
performance  objectives,  as  may  be  applicable  to  projected  on-line  test  and 
monitoring  usage? 

•  Fault  detection 

•  False  alarm  rate  of  occurrence 

•  False  alarm  percentage  of  indicated  faults 

•  Fault  isolation  resolution 

•  Fault  isolation  percentage 

•  MTTR 

•  Mean  BIT  running  time 

•  Percentage  of  functions  BIT  tested 

•  Percentage  of  all  functions  monitored 

•  Consideration  of  complexity  of  BIT  impact  upon  overall 
system  complexity  (and  hence  reliability). 

d.  Have  all  proposals  been  reviewed  for  adequacy  in  meeting  the 
following  design  for  accessibility  and  monitoring  requirements? 

•  Hardware  modularization 

•  Software  modularization 

•  External  test/control  points 

0  Emulation  capability  (ICE) 


Table  4.  Conceptual  Proposal  Evaluation  Checklist 
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DEMONSTRATION  AND  VALIDATION  PHASE 


2.0  This  phase  of  the  acquisition  provides  increasingly  expanded 
definition  of  the  alternatives  selected  at  the  conclusion  of  the  concept 
exploration  phase.  Requirements  for  on-line  test  and  monitoring 
capability,  such  as  operational  status  monitoring,  self-test  and  fault 
tolerance,  also  take  shape  in  the  process.  Integration  of  this  capability 
with  all  other  allied  design  engineering  specialties,  such  as  R&M  and 
accessibility,  is  made  to  optimize  the  overall  performance  and  cost 
aspects  of  these  interrelated  disciplines.  The  flow  of  activity  for  this 
phase  is  depicted  in  Figure  3  at  the  end  of  this  section. 


2.1  AUTHORITY  TO  PROCEED  (POD I  5000.2,  SECNAVI  5000.1).  SECNAV  approval 
of  the  DCP  constitutes  authority  for  the  acquisition  to  proceed  into  the 
Demonstration  and  Validation  Phase.  In  anticipation  of  such  authority, 
specialty  plans  to  be  produced  by  the  contractors,  such  as  the  ILSP  and 
the  T&E  management  plans,  should  be  identified.  Since  no  current 
compliance  document  exists  for  invoking  a  testability  plan,  this 
requirement  should  be  specified  by  a  Statement  of  Work  task  statement. 


2.2  ISSUE  DEVELOPMENT  CONTRACT(S)  (DODD  5000.1,  SECNAVI  5000.1). 

Cont i nuation  of  competitive  development  into  the  advanced  development 
stages  should  be  pursued  in  interest  of  getting  the  best  product  at  the 
best  price.  As  such,  on  receipt  of  the  SECDEF  authority  to  proceed, 
contracts  are  issued  to  those  contractors  whose  proposals  were  judged  as 
having  capability  to  solve  mission  need  economically.  Consideration 
should  be  made  for  continuing  as  a  single-source  acquisition  only  where 
urgency  of  need,  or  the  absolute  excellence  of  one  proposal  is  evident 
over  all  others.  The  contract  should  provide  for  refinement  of  the 
technical,  costs,  and  schedule  efforts  through  requirements  to  update  the 
system  specification,  as  well  as  for  system-level  prototype  testing, 
initial  design  reviews,  and  the  initiation  of  Type  B  Prime  Item 
Specification  drafts. 


2.3  SYSTEM  DEFINITION,  TRADE-OFFS  HARDWARE/SOFTWARE  PRELIMINARY  DESIGN 
(PRELIMINARY  SYSTEM  SPECIFICATION,  MIL-STD-1366) .  The  principal 
activities  of  the  contractor(s)  during  the  Demonstration  and  Validation 
Phase  are  to  verify  concept(s)  and  to  gather  data  for  use  in  defining  the 
system  design  in  the  Full  Scale  Development  Phase.  System  definition  is 
continually  refined,  first  through  trade-off  of  pure  design  factors  based 
on  experience  and  historical  data,  to  arrive  at  a  design  model. 
Subsequent  trade-offs  will  include  cost  and  logistic  support  factors; 
further  refined  by  Logistic  Support  and  Level  of  Repair  (LOR)  Analyses  to 
produce  increasing  definition  of  qualitative  and  quantitative  logistics 
and  maintenance  support  requirements .  Refinement  in  predictions  of 
logistic  support  costs  in  funds  and  other  resources  at  all  levels  of 
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repair,  is  also  obtained  from  these  analyses.  Resultant  on-line 
testability  and  monitoring  application  may  provide  the  optimum  system 
configuration,  depending  upon  operational  requirement  peculiarities. 
Additionally,  on-line  concepts  such  as  fault  monitoring  and  detection, 
system  status  monitoring,  and  fault  tolerance,  should  be  considered  in  the 
trade-offs  and  logistics  analyses  to  arrive  at  the  most  cost-effective 
technically  adequate  solution. 


2.4  SYSTEM  REQUIREMENTS  REVIEW  (HIL-STD-1521,  MIL- STD-499).  The  SRR  is 
conducted  when  system  functional  requirements  become  evident  as  a  result 
of  demonstration  and  validation.  The  purpose  of  the  SRR  is  to  review 
progress  and  direction  of  the  contractor's  system  engineering  management 
activity,  particularly  the  status  of  system/subsystem  functional 
identification  and  allocation  of  performance  and  design  requisites  to  the 
identified  functional  subsystems.  The  SRR  should  ascertain  the 
sufficiency  of  testability  in  the  proposed  system  to  assure  that  proper 
balance  and  compatibility  is  planned  between  the  proposed  on-line 
functions  (monitoring/self-test/self-maintenance) ,  and  the  off-line 
functions  (manual  test/semiautomatic  test  and  ATE).  The  minimization  of 
false  monitoring  and  testing  indications  and  ambiguities  should  be  a  goal 
to  be  validated  in  this  phase.  Table  5  contains  a  checklist  of  the  major 
on-line  test  and  monitoring  items  to  be  checked  at  the  SRR. 


2.5  UPDATE  PRELIMINARY  SYSTEM  SPECIFICATION  (MIL-STD-490) .  The  pre- 
limi nary  system  specification  is  updated  by  the  contr actor (s)  to  reflect 
the  changes  made  during  the  demonstration  and  validation  efforts. 

In  reviewing  the  updated  draft  of  the  system  specification,  the  program 
office  should  insure  that  on-line  test  and  monitoring  functions  have  been 
identified,  that  they  contribute  to  the  mission-need  solution,  and  that 
performance  characteristic  values  and  parameters  are  determined. 

Baselines  may  be  established  at  any  time  that  it  may  be  necessary  to 
define  a  formal  departure  point  for  the  control  of  future  changes  in 
performance  or  design.  The  functional  baseline  is  most  commonly  defined 
by  a  system  specification,  such  as  the  preliminary  system  specification 
prepared  in-house.  For  purposes  of  this  document,  the  functional  baseline 
is  set  when  the  contractors'  update  of  the  preliminary  specification  is 
approved.  All  subsequent  system  performance  or  design  changes  in 
hardware,  software  or  firmware  (including  the  system  specification),  must 
be  handled  as  configuration  management  changes. 


2.6  PROTOTYPE/PAPER  STUDY  TESTING  (DODD  5000.3).  Early  test  and  evalua¬ 
tion  of  on-line  test  and  monitoring  concepts  lends  itself  readily  to  the 
following  means  of  testing: 

•  Paper  Studies.  Review  of  historical  testing  reports  conducted 
on  similar  previous  concepts,  can  be  used  to  determine  and 
clarify  the  basic  T&E  requirements  and  structure. 
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Contractor  Design  for  Testability  (DFT)  activities  to  be  checked 
during  the  SRR  should  include  the  following  items  as  applicable: 

a.  Within  the  basic  requirement  of  fulfilling  or  solving  mission 
needs,  do  the  operational  requirements  tend  toward  the  application  of  on¬ 
line  test  and  monitoring  techniques.... 

•  as  the  sole  testing  capability? 

•  in  conjunction  with  off-line  testing? 

•  not  at  all? 

b.  Has  sufficient  study/analysis  been  accomplished  to  ascertain  an 
optimum  mix  of  on-line  and  off-line  capability  to  determine  system  "state- 
of -health"  and  to  isolate  or  correct  faults? 

c.  Are  on-line  test  and  monitoring  functions  represented  in  system 
functional  flow  analyses? 

d.  Do  trade-off  studies  address  system  on-line  test  and  monitoring 
functions  in  hardware/f irmware/sof tware  uses? 

e.  Do  system  cost  and  effectiveness  analyses  reflect  consideration 
of  on-line  test  and  monitoring  variances  for  both  Design  to  Cost  (DTC)  and 
Life  Cycle  Cost  (LCC)  as  follows: 

•  On-line  versus  off-line  testing/monitoring  capability 

•  BIT  versus  no  BIT 

•  Fault  tolerance  capability  versus  manual  repair  methods 

•  Self  calibration  capability 

f.  Have  analyses/trade  studies  been  conducted  to  determine  that 
integration  of  on-line  test  and  monitoring  design  principles  will  support 
the  other  specialty  disciplines  such  as  reliability  and  maintainability? 

g.  Has  on-line  test  and  monitoring  capability  been  successfully 
initiated  in  preliminary  designs  at  the  system/subsystem  levels? 

h.  Have  test  plans  and  procedures  included  requirement  to 
demonstrate  the  planned  on-line  test  and  monitoring  concepts? 


Table  5.  System  Requirements  Review  Checklist 
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•  Existing  Systems.  The  use  of  similar  existing  systems,  modified 
as  necessary,  should  be  considered  in  early  estimations  of 
operational  effectiveness  and  suitability. 

•  Hardware  Prototyping.  The  use  of  a  breadboard/brassboard 
fabrication  should  be  considered.  Construction  should  emulate 
the  operational  system  performance  objectives.  Prototyping  is 
especially  applicable  to  demonstrating  and  validating  on-line 
test  and  monitoring  circuits  and  software  routines,  as  well  as 
the  functional  compatibility  of  same  with  other  disciplines. 


2.7  PREPARE  SOW  FOR  FULL  SCALE  DEVELOPMENT  (FSD) .  When  it  is  advanta¬ 
geous  to  continue  the  competition  into  full  scale  development,  a  Request 
For  Proposal  (RFP)  is  prepared.  This  RFP  is  issued  during  the  validation 
phase  to  permit  timely  response  by  contractors  so  the  selection  process 
can  determine  the  best  one  or  two  alternatives/contractors  to  proceed  into 
FSD.  When  further  competition  is  not  necessary  and  approval  is  granted  to 
continue,  the  FSD  effort  will  be  put  out  on  contract.  In  either  case,  the 
statement  of  work  requirements  should  be  concisely  defined  and  stated  to 
insure  that  derived  system  performance  parameters,  including  on-line  test 
and  monitoring  criteria,  are  translated  into  1 og istical ly  supportable, 
demonstrable,  and  cost-effective  performance  requirements  for 
Configuration  Items  (CIs).  Table  6  contains  task  statements  of  on-line 
test  and  monitoring  considerations  for  the  Full  Scale  Development  SOW. 


2.8  PREPARE  DEVELOPMENT  SPECIFICATION  DRAFTS  (MIL-STD-490,  MIL-STD-483) . 

As  the  performance  parameters  for  a  needed  capability  are  refined  in  the 
system  specification,  they  are  translated  into  performance  requirements 
for  lower  tier  Configuration  Items.  Such  derived  requirements  should  be 
specified  in  the  SOW  for  reflection  in  contractor-produced  Development 
Specifications.  These  requirements  should  be  stated  in  sufficient  detail  . 
to  insure  positive  tracking  into  Cl  hardware  and  software  products. 

In  review  of  these  contractor-produced  specifications  drafts,  the  program 
office  should  insure  that  SOW  requirements  are  met. 


2.9  SYSTEM  DESIGN  REVIEW  (SDR)  (MIL-STD-1521,  NAVMATI  4130.1).  This 
review  is  conducted  when  the  system  requirements  and  design  approach  are 
defined.  Its  purpose  is  to  evaluate  the  optimization,  traceability, 
correlation,  completeness,  and  the  risk  of  the  allocated  requirements, 
including  corresponding  test  requirements.  This  review  includes  all 
analyses,  trade-offs  and  other  studies,  from  which  total  system 
requirements  evolved.  Table  7  contains  a  checklist  of  on-line  test  and 
monitoring  items  for  consideration  at  SDR. 


The  following  Design  for  Testability  (DFT)  task  statements  are 
directed  at  design  requirements  for  on-line  test  and  monitoring 

architecture,  to  be  reflected  in  Development  Specifications.  Design  shall 
consider  actual  noise  environment  in  interest  of  suppressing  false  alarms. 

a.  The  design  goal  for  Percentage  of  Faults  Detected  (i.e.,  fault 

coverage)  shall  be  _  percent  or  greater  of  the  total  known  possible 

fault  population. 

b.  The  contractor's  design  for  Percentage  of  False  Alarms  to  total 

false  indications,  shall  not  exceed  _  percent.  False  alarm 

determination  may  also  be  stated  as  a  discrete  number,  as  follows: 

o  The  acceptable  number  of  false  alarms  shall  not  exceed 
per  _  hours  of  operation. 

c.  The  contractor's  design  for  Mean  Fault  Detection  T-w  m^s*  nct 

exceed  _  (selected  units  of  time)  to  detect  or  indicate  thp  presence 

of  a  fault. 

d.  The  contractor  design  for  Mean  Bit  Pur.n’nq  T  •  •  • 

that  a  maximum  of  _  units  of  time  is  needed  to  w-  *  ,  ■.:■■■■■ 

fault  is  in  fact,  a  failure. 

e.  The  contractor  shall  design  Test  v>:r  •  .  "■ 

(percentage  of  all  functions  which  are  ^on-t  • :  • 

greater  than _ %. 

NOTE:  A  comprehensive  application  of  5IT  tern-  •  •  .  •  ; 

the  utilization  of  lower-skilled,  operating /mjirten 
should  be  a  cons iderat ion  in  decisions  on  the  e»t-n*  (  , 

as  its  design.  Hence:  design  of  the  fault  c  s  <  *  >- •  -  *  •  •  .  •  > 

(printed  or  displayed)  shall  be  appropriate  for  the  s*  ■  ’  ’  '•  .•  '  • 

operator /user . 

f.  The  contractor  shall  design  system,  subsystem  and  con*  ’ qur at • m 

items  to  operator/user  skill  levels _ ;  and  maintenance  skill  leve’ 

_  at  the  _  level  of  maintenance. 

g.  The  contractor  shall  design  for  an  Operational  Availability  of 

no  less  than  _  percent. 

h.  The  contractor  shall  determine  adequate  memory  capacity  for  all 
testability  requirements,  including  on-line  test  and  monitoring  functions. 


Table  6.  Full  Scale  Development  SOW  Task  Statements 
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2.10  SUBMIT  FSD  PROPOSAL  (OOP!  5000.2,  SECWAV1  5000.1).  Technical  and 
cost  proposals  for  the  Full  Scale  Development  (FSD)  effort  are  prepared  by 
the  contractors  selected  to  proceed  into  FSD.  These  proposals  are  based 
on  RFP  requirements  for  defined  design  and  development,  fabrication  and 
testing  of  the  system,  including  support  equipment  and  documentation. 
Contractors  will  propose  how  they  intend  to  meet  the  qualitative  vaues  of 
the  specialty  disciplines,  including  on-line  test  and  monitoring  as 
specified  in  the  RFP. 


2.11  REVIEW  AMD  SELECT  FSD  PROPOSALS  (DODD  5000.1,  OPNAVI  5000.1).  The 

procedures  for  the  review  and  selection  of  FSD  proposals  parallels  those 
for  the  conceptual  stage.  The  increased  level  of  definition  in  technical 
and  support  inputs,  design  characteristics,  DTC/LCC  results,  and 
validation  results  in  turn  require  increased  consideration  in  proposal 
evaluations.  When  total  demonstration  and  validation  effort  is  evaluated 
and  decision  is  made  as  to  which  proposal(s)  should  be  selected  for  full 
scale  development,  action  is  taken  to  update  the  DCP.  Table  8  contains  a 
checklist  of  proposal  evaluation  criteria,  as  pertains  to  on-line  test  and 
monitoring  functions. 


2.12  UPDATE  DCP  (DODI  5000.2,  OPNAV  5000.24).  Update  of  the  Milestone  I 
DCP  is  accomplished  in  preparation  for  the  Milestone  II  Decision.  On-line 
test  and  monitoring  concepts,  goals  and  thresholds  are  updated,  based  on 
increasing  refinement  and  definition  resulting  from  trade  studies, 
analyses,  and  validation  actions. 

2.12.1  DCP  Approval  (DODD  5000.1,  OPNAVI  5000.46).  Prior  to  Milestone 
II,  the  DCP  is  reviewed  by  the  Defense  Systems  Acquisition  Review  Council 
(DSARC)  in  preparation  of  recommended  action  to  the  SECDEF.  Approval  by 
the  SECDEF  sets  Milestone  II  and  ends  the  Demonstration  and  Validation 
Phase  of  the  acquisition. 
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a.  Has  on-line  test  and  performance  monitoring  capability  been 
given  consideration  appropriate  to  the  mission,  support,  and  functional 
analyses,  for  application  in  the  design  and  performance  of  the  proposed 
system? 


b.  Has  this  capability  been  reflected  in  the  updated  system 
specification? 

c.  Are  on-line  test  and  monitoring  requirements  translated  to  the 
Cl  Development  Specification  Drafts? 

d.  Do  trade  studies  and  analyses  reflect  the  effects  of  on-line 
test  and  monitoring  capability  on  R&M,  testability,  logistics  support,  and 
level  of  repair  in  technical  as  well  as  cost  terms? 

e.  Have  all  risks  associated  with  the  application  of  performance 
monitoring  or  on-line  test  been  identified  and  assessed,  including  the 
potential  impact  on  reliability,  maintainability,  and  operator  skill  level 
requirements? 

f.  Has  the  final  combination  of  operations,  maintenance,  logistics 
and  test  requirements  affected  the  overall  program  concept  in  any  negative 
manner  which  is  excessively  detrimental  to  the  fulfillment  of  the  overall 
mission  needs? 

g.  Are  operational  availability,  performance,  overall  support- 
ability  and  LCC  of  the  proposed  system  all  within  prescribed  bounds? 


Table  7.  System  Design  Review  Checklist 
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monitoring 

evaluation 


following  items  of  evaluation 
requirements  could  have  impact, 
results . 


in  wh  i 
should 


a.  Has  sufficient  consideration  been  given 
system  specification  requirements  to  lower  tier  (Cl) 


ch 

on-1 

ine  test 

and 

be 

cons 

idered  in 

the 

to 

the 

allocation 

of 

requirements? 

b.  Has  consideration  and  application  of  this  capability  been  made 
when  analysis  indicates  increased  operational  efficiency  or  ICC  cost 
reduction  is  probable? 

c.  Has  sufficient  depth  of  definition  been  given  to  the  performance 
requirements  stated  in  the  Cl  development  specifications? 

d.  Have  refined  costs  (covered  partly  in  "b"  above)  and  refined 
schedule  estimates  been  made,  and  are  they  realistic? 

e.  Do  the  breadboard  test  results  of  the  on-line  test  and 
monitoring  functions  sufficiently  validate  the  requirement? 


f.  Are  the  level  and  mix  of  on-line  test  and  monitoring 
applications  optimized  for  the  mission  need? 

g.  Are  any  unique  problems  or  risks  evident  in  the  use  of  this 
capability? 

h.  Have  all  interface  requirements  (functional  and  physical)  been 

adequately  designed?  Are  interface  control  procedures  adequate? 

i.  Have  the  risks  that  on-line  test  and  monitoring  impose  on  other 
disciplines  such  as  reliability  and  maintainability  been  fully  considered? 

j.  Is  consideration  given  in  configuration  management  procedures 
for  tracking  and  controlling  the  effect  that  system  design  changes  or 
changes  in  other  specialized  disciplines  may  have  on  the  performance 
monitoring  requirements  and  architecture? 

k.  Have  the  offerors  identified  all  on-line  test  and  monitoring 

parameters  subject  to  having  effect  in  LSAs?  Also,  do  logistics  plans 

provide  for  refinement  of  these  parameters  in  LSA  iterations? 

l.  Does  the  proposed  on-line  test  and  monitoring  structure  fully 

comply  with  the  RFP,  and  therfore,  mission  need  requirements? 


Table  8.  Full  Scale  Development  Proposal  Evaluation  Checklist 
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SECTION  3 


FULL  SCALE  DEVELOPMENT  PHASE 


3.0  During  this  phase  of  the  acquisition,  a  pre-production  prototype  of 
the  system,  along  with  documentation  needed  to  produce  the  system  for  the 
operational  inventory,  is  designed,  developed,  fabricated  and  tested. 
Detailed  design  specif ications  and  drawings  are  readied  in  preparation  for 
the  increasingly  critical  design  reviews  by  program  managment  office  and 
other  interested  personnel,  and  specifications  and  drawings  are  examined 
in  ever-increasing  detail  to  assure  that  the  configurations  are  in 
accordance  with  the  design  requirements.  The  preliminary  and  critical 
design  reviews  afford  the  last  opportunity  to  consider  adequacy  of  the  on¬ 
line  test  and  monitoring  applications.  The  flow  of  activity  for  this 
phase  is  depicted  in  Figure  4  at  the  end  of  this  section. 


3.1  RECEIVE  AUTHORITY  TO  PROCEED  (DODI  5000.2,  SECNAVI  5000.1).  As  at 

the  previous  milestone,  the  SECDEF  approval  of  the  DCP  constitutes 
authority  to  proceed  into  FSD. 


3.2  ISSUE  FSD  CONTRACT.  The  technical  portion  of  the  FSD  contract  is 
constructed  from  thi  SOW  prepared  in  the  Demonstration  and  Validation 
Phase.  It  is  issued  to  the  one  or  two  contractor(s)  selected  to  pursue 
full-scale  development  of  their  proposed  systems. 


3.3  CONTRACTOR  TRADE-OFF,  INITIAL  DEVELOPMENT,  PRELIMINARY  DESIGN 
ACTIVITY  (MIL-STD-i3S8)T~  The  ever-increasing  formalization  of  the  design 
process  in  this  phase  calls  for  continuous  LSA  refinement  and  updating. 
This  provides  in-depth  design  feedback  of  the  various  engineering 
specialty  areas  which  have  impact  on  testability  and  the  ability  to 
monitor  system  performance.  Examples  are:  (a)  the  Failure  Mode  and 
Effects  Analysis  (MIL-STD-785) ,  and  (b)  Mean  Time  to  Repair  data  (MIL-STD- 
470),  both  required  in  the  quantification  of  test  and  monitoring  design 
factors  such  as  fault  detection  and  isolation  times,  false  alarm  data  and 
BIT  running  times.  These  LSA-derived  design  factors  are  "traded  off"  with 
cost  and  time  criteria. 


3.4  FINALIZE  HARDWARE/SOFTWARE  Cl  DEVELOPMENT  SPECIFICATIONS  (MIL-STD- 
490,  MIL-STD-483).  TTTe  draft  Development  Specification  for  each 

configuration  item  is  updated  by  the  contractor(s)  based  on  comments 
received  from  the  program  management  office  (PMO)  review,  and  any  PMO- 
negotiated  changes  resulting  from  on-going  LSA  refinement.  These 
finalized  specifications  are  forwarded  to  the  PMO  for  authentication, 
establishing  the  performance  and  test  criteria  on  which  the  program  is 
design  and  developed. 
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3.5  PRELIMINARY  DESIGN  REVIEW  (PDR)  (MIL-STD-1521).  The  PDR  is 

conducted  to  document  and  evaluate  the  selected  design  approach,  technical 
adequacy,  and  risk  resolution  on  cost  and  schedule  bases  at  the 
configuration  item  level.  The  review  will  include  those  design  products 
in  which  on-line  test  and  monitoring  requirements  have  impact.  Table  9 
contains  a  checklist  of  on-line  test  and  monitoring  items  for 
consideration  during  PDR. 


3.6  ON-GOING  DT&E  (DODD  5000.3,  SECNAV  5000.1).  Tests  and  evaluations 
in  the  F^D^  including  development  qualification  testing,  are  conducted  to 
reduce  the  risk  of  the  system  or  equipment  being  unable  to  meet  specified 
performance,  or  the  risk  of  being  incapable  of  employment  in  the  intended 
operational  environment.  Testing  and  evaluation  plans,  procedures  and 
results  are  reviewed  during  the  engineering  reviews  to  insure  that  design 
and  performance  requirements,  including  on-line  test  and  monitoring 
requirements,  meet  specif ication  and  thus  support  milestone  decisions. 


3.7  DETAIL  DEVELOPMENT  DESIGN.  On  successful  completion  of  the  PDR,  and 
as  the  system  engineering  process  continues,  detail  design  for  hardware 
and  software  elements  evolve  from  the  preliminary  design  state. 
Progressive  ISA  and  LORA  are  used  to  derive  a  definitive  support  posture 
for  the  resultant  products.  As  may  be  necessary,  prototype 
hardware/software  items  are  fabricated  to  test/verify  the  on-line  test  and 
monitoring  functions  (among  the  other  specialty  engineering  requirements). 


3.8  DRAFT  PRODUCT  HARDWARE /SOFTWARE  SPECIFICATIONS  (MIL-STD-490,  MIL- 
STD-483TI  The  Cl  Product  Specif ication(s)  is  initiated  as  detail  design 
data  evolves  as  a  result  of  the  continuing  system  engineering  process.  On 
completion  of  the  specif ication(s)  and  in  preparation  for  the  Functional 
Configuration  Audit  and  the  Critical  Design  Review,  draft  copies  are 
forwarded  to  the  P MO  for  review. 


3.9  FUNCTIONAL  CONFIGURATION  AUDIT  (FCA)  (MIL-STD-1521).  The  FCA  is  a 

formal  examination  of  the  Configuration  Item  (Cl)  test  data.  Test  results 
and  procedures  (or  adequate  analyses  or  simulation  as  may  be  accepted), 
are  reviewed  to  verify  achievement  of  the  Cl's  performance  specified  in 
the  Development  Specification. 


3.10  CRITICAL  DESIGN  REVIEW  (CDR)  (MIL-STD-1525) .  The  CDR  is  conducted 
prior  ~to  fabrication/production  release  to  insure  that  the  Development 
Specification  performance  requirements  are  satisfied  by  the  detail  design 
solutions  reflected  in  the  draft  Product  Specification  and  engineering 
drawings.  During  the  CDR,  the  contractor  is  also  required  to  show 
compatibility  of  functional  and  physical  interfaces,  both  internal  and 
external,  to  the  Cl.  Table  10  contains  a  checklist  of  on-line  test  and 
monitoring  items  of  interest  during  the  CDR. 
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a.  Equipment  Configuration  Items 


1.  Do  the  performance  characteristics  for  testability,  as 

authenticated  in  the  system  specification,  translate  into 
Cl-level  performance  requirements? 

2.  Have  trade-studies  regarding  reliability,  maintainability 
and  testability  been  completed  with  results  incorporated 
into  system  design? 

3.  Has  BITE  versus  Electronic  Test  Equipment  (ETE)  study  been 
performed,  and  results  employed  in  design? 

4.  Has  provision  for  self-test  firmware  been  made  in 

equipment/performance  design?  Have  provisions  been  made  for 
micro-program  diagrams  and  descriptions  of  algorithms  for 
instruction  translation,  fabrication  and  packaging 
(integration  technology),  and  special  equipment  and  support 
software  needed  to  develop  and  test  the  self-test  firmware? 

b.  Support  Equipment 

1.  Has  the  optimal  Support  Equipment  arrangement  consisting  of 
BITE  and/or  separate  test  equipment  (ETE/ATE)  been  arranged? 

c.  Design  for  Reliability,  Maintainability  and  Testability 

1.  Review  the  contractor's  planned  actions  for  adequacy  when 
predictions  indicate  that  specified  requirements  will  not  be 
attained. 

2.  Review  the  adequacy  of  the  testability  planning  data  to 
insure  that  the  indicated  self-test  and  performance 
monitoring  design  principles  provide  adequate  and  mature 
overall  testability  and  monitor  capability  design.  Verify 
that  contractor  design  engineers  and  engineering  management 
follow  this  planning  data. 

3.  Insure  that  human  factor  considerations  that  are  pertinent 
to  self-test  and  performance  monitoring  have  been  given 
adequate  consideration.  Have  factors  of  display  techniques, 
control  and  data  entry  devices,  as  well  as  status,  error  and 
data  printout  messages  (as  appropriate)  all  been  considered, 
individually  and  in  concert,  as  to  their  usability  from  the 
human  interface  standpoint? 


Table  9.  Preliminary  Design  Review  Checklist 
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For  each  Cl,  review  the  adequacy  of  the  detail  design  of  the  draft 
Product  Specification,  for  compatibility  with  the  Development 
Specification  performance  requirements. 

a.  Review  the  adequacy  of  the  detailed  design  in  the  areas  of 
reliability,  maintainability  and  testability.  Do  all  details  of  self-test 
or  performance  monitoring  design  functions  satisfy  the  requirements  of  the 
Development  Specification  while  having  no  negative  impact  upon  system 
reliability,  maintainabi 1 ity,  etc. 

b.  Similarly,  review  self-test  routines  (diagnostic  procedures) 
contained  within  firmware  to  be  provided  for  the  system,  for  adequacy  of 
detail  design  and  compliance  with  the  other  engineering  specialty 
disciplines. 

c.  Identify  Design  For  Testability  (DFT)  criteria  provided  by  the 
Full  Scale  Development  SOW  task  statements,  to  insure  that  criteria  have 
been  incorporated. 

d.  Identify  any  unique  self-test/performance  monitoring  procedures 
required  for  each  Cl  when  in  operational  use,  and  evaluate  their  effect  on 
system  test  and  maintenance  concepts.  Assure  that  the  system  is  optimized 
from  the  performance  monitoring  point  of  view. 

e.  Review  performance  monitoring  or  BIT  for  adequacy  of  coverage  as 
well  as  compatibility  with  overall  mission  requirements.  Does  it 
interface  well  with  the  Cl  primary  functions  without  degradation  of 
performance,  while  being  adequate  in  the  human  factors  viewpoint  as  well? 


Table  10.  Critical  Oesign  Review  Checklist 


3.11  FINALIZE  PRODUCT  SPECIFICATIONS  (HIL-STD-490,  HIL-STD-483).  The 
draft  Product  Specification  for  each  configuration  item  is  updated  by  the 
contractors )  based  on  comments  received  from  the  program  management 
office  review,  as  well  as  any  PMO-negotiated  changes  resulting  from  on¬ 
going  LSA  refinement.  As  they  are  finalized,  these  specifications  are 
forwarded  to  the  PMO  for  review.  When  authenticated,  they  become  the 
basis  for  the  engineering  drawings  on  which  production  is  based. 
Authentication  of  each  Product  Specification  establishes  the  Product 
Baseline  for  the  Cl.  A  limited  number  of  the  items  may  be  produced  for 
qualification  testing  after  authorization  of  the  Product  Specification  has 
been  made. 


3.12  UPDATE  DCP  (POD I  5000.2,  OPNAV  5000.46).  In  preparation  of  the 
Milestone  III  decision  to  proceed  into  the  Production  Phase,  the  previous 
(Milestone  II  DCP)  is  finalized  with  updated  goals  and  thresholds  based  on 
the  detailed  and  firm  "build-to"  specification  and  engineering  drawing 
products  of  the  FSD  phase. 

3.12.1  DCP  Approval  (DODD  5000.1,  OPNAVI  5000.46).  As  previously 
stated,  approval  of  the  DCP  constitutes  authority  to  proceed  to  the  next 
phase  of  acquisition,  in  this  case,  Production  and  Deployment. 
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SECTION  4 


PRODUCTION  AND  DEPLOYMENT  PHASE 


4.0  During  the  Production  and  Deployment  phase,  the  system,  including 
support  and  training  equipment,  is  produced  for  operational  use. 
Production  units  are  selectively  acceptance-tested  by  Government 
personnel.  Operational  Test  and  Evaluation  is  conducted  under  actual 
operational  conditions  in  preparation  for  delivery  to  the  operational 
forces.  All  verification  of  hardware/sof tware  compliance  with  final 
specification  requirements  should  be  completed  during  this  phase.  System 
elements  are  integrated  into  a  complete  system  to  be  tested  for  compliance 
with  the  system  specification.  Finally  the  configuration  management 
process  is  continued  into  deployment  for  design  feedback  as  may  be  made 
necessary  by  engineering  changes.  The  flow  of  activity  for  this  phase  is 
depicted  in  Figure  5  at  the  end  of  this  section. 


4.1  ST  ART  PRODUCT I ON .  Following  the  Milestone  III  decision,  the 
commitment  for  production  is  made  contractual  based  on  the  authenticated 
"build-to"  Production  Specif ication(s) . 


4.2  PHYSICAL  CONFIGURATION  AUDIT  (PCA)  (MIL-STD-1521 ,  MIL-STD-483) .  The 

function  of  the  audit  team,  is  to  verify  that  the  as-built  version  of 
First  Article  CIs  conform  to  the  Production  Specifications.  It  also 
includes  an  audit  of  specifications  and  techical  data  such  as  engineering 
drawings  for  hardware  items  and  users'  manuals  for  software  items. 
Acceptance  test  procedures  are  reviewed  for  adequacy  during  the  PCA. 
Finally,  engineering  release  documents  and  quality  control  records  are 
checked  against  the  as-built  configuration  of  the  article  and  its  related 
documentation  to  insure  late  design  changes  are  incorporated.  This  is 
especially  important  in  self-testing  and  monitoring  functions  in  the 
interest  of  eliminating  false  monitoring  and  testing  ambiguities. 


4.3  ACCEPTANCE  TESTING.  Production  acceptance  tests  are  conducted  in 
accordance  with  the  quality  assurance  provisions  in  Section  4  of  each 
Configuration  Item  Product  Specification.  Val idation/verif ication  of  on¬ 
line  test  and  monitoring  performance  should  be  considered  as  a  special 
acceptance  test  requirement. 


4.4  OPERATIONAL  TEST  AND  EVALUATION  (OT&E).  As  initial  CIs  are  placed 
into  service,  OT&E  takes  on  greater  significance.  Within  the  OT&E  effort, 
service  tests  of  reliability  and  maintainability  are  conducted  to  verify 
related  quantitative  data.  The  closely  allied  'testability'  discipline 
should  also  be  included  in  OT&E  planning  to  require  verification  of 
applicable  on-line  test  and  monitoring  quantitative  requirements. 
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4.4.1  Design  Feedback.  Any  discrepancies  discovered  in  OT&E  that  require 
design  changes  must  be  fed  back  through  the  contractor's  configuration 
management  system  to  maintain  configuration  control  of  the  item/system. 
Resultant  impact  upon  on-line  test  and  monitoring  circuits/structure  must 
be  considered  to  insure  minimization  of  false  alarms  in  test  and 
monitoring  performance. 


4.5  OPERATIONAL  USE.  The  operational  forces  also  conduct  suitability 
tests  oriented  to  tactical  use  on  units  received  directly  from  production. 

4.5.1  Engineering  Changes.  Any  required  engineering  changes  are 
similarly  entered  into  tne  engineering  change  process  for  design  feedback 
to  insure  that  impact  on  all  functional  and  performance  areas  is 
considered. 
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List  of  Acronyms 


ATE 

Automatic  Test  Equipment 

BIT 

Built-In-Test 

BITE 

Built  In  Test  Equipment 

CDR 

Critical  Design  Review 

Cl 

Configuration  Item 

CM 

Configuration  Management 

CMP 

Configuration  Management  Plan 

DCP 

Decision  Coordination  Paper 

OFT 

Design  For  Testability 

DoD 

Department  of  Defense 

DoDD 

Department  of  Defense  Directive 

DoD  I 

Department  of  Defense  Instruction 

DSARC 

Defense  Systems  Acquisition  Review  Council 

DTC 

Design  To  Cost 

DT&E 

Development  Test  and  Evaluation 

ETE 

Electronic  Test  Equipment 

FCA 

Functional  Configuration  Audit 

FSD 

Full  Scale  Development 

IC 

Integrated  Circuit 

ILS 

Integrated  Logistic  Support 

ILSP 

Integrated  Logistic  Support  Plan 

ICC 

Life  Cycle  Cost 

LOR 

Level  Of  Repair 

LORA 

Level  of  Repair  Analysis 

LSA 

Logistics  Support  Analysis 

LSI 

Large  Scale  Integration 

MENS 

Mission  Element  Need  Statement 

MIL-HDBK 

Mi  1 itary  Handbook 

MIL-STD 

Military  Standard 

MSI 

Medium  Scale  Integration 

MTFL 

Mean  Time  to  Fault  Locate 

MTTR 

Mean  Time  To  Repair 

List  of  Acronyms,  Continued 


NAVMAT 

Naval  Material 

0MB 

Office  of  Management  and  Budget 

OT&E 

Operational  Test  and  Evaluation 

PCA 

Physical  Configuration  Audit 

PDR 

Preliminary  Design  Review 

PM 

Program  Manager 

PMO 

Program  Manager  Office 

RFP 

Request  For  Proposal 

R&M 

Reliability  and  Maintainability 

SDR 

System  Design  Review 

SECDEF 

Secretary  of  Defense 

SEMP 

System  Engineering  Management  Plan 

SOW 

Statement  Of  Work 

SRR 

System  Requirement  Review 

SYSCOM 

Systems  Command 

TO 

Technical  Document 

TEMP 

Test  &  Evaluation  Master  Plan 

T&E 

Test  and  Evaluation 

VLSI 

Vary  Large  Scale  Integration 
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